Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Framework: Add queried-data state helper utility #6395

Closed
wants to merge 5 commits into from

Conversation

aduth
Copy link
Member

@aduth aduth commented Apr 24, 2018

Related: #6180
Related: #6360

This pull request seeks to work toward a generalized solution for queried / paginated data resources. It is implemented as an independent (but currently bundled) set of state helpers, including reducer, selectors, and action creators for operating on queried data. Currently, only the categories data has been refactored, and it does not currently take advantage of pagination (as this is not used by the Categories block). Further, as discovered in #6360, some additional work is required for memoization of "same" (but not strictly equal) query fulfillment.

The hope is that each resource (core-data state key) can simply reuse the existing helpers from queried-data. For terms, this looks like:

export const terms = onSubKey( 'taxonomy' )( queriedDataReducer );

Some additional helpers may be required to expand this out to other state like media, particularly for ensuring that the reducer only operates on relevant queried data. For terms this is handled automatically through the onSubKey higher-order reducer creator, where state is only tracked for actions which include a taxonomy property. Likewise, we may create other higher-order reducers which test for the presence of some resource-specific marker, to be applied as an extension of the baseline receiveQueriedItems action creator (e.g. ifActionMatches( ( action ) => action.queried === 'media' )).

The state shape is normalized, with queries tracked as stable keys, both to avoid storing redundant data, and to avoid duplicate tracking for equivalent queries:

Example:

{
	"terms": {
		"categories": {
			"items": {
				"1": {
					"id": 1,
					"name": "Uncategorized"
				},
				"2": {
					"id": 2,
					"name": "Cats"
				}
			},
			"queries": {
				"s=cat": {
					"itemIds": [ 1, 2 ],
					"requestingPageByPerPage": {}
				},
				"s=cats": {
					"itemIds": [ 2 ],
					"requestingPageByPerPage": {}
				}
			}
		}
	}
}

Testing instructions:

This should only impact the Categories block, which is the sole consumer of categories data. Verify that there are no regressions in the display of categories block, including the "Loading" placeholder and ultimately the display of categories.

Ensure unit tests pass:

npm test

@aduth aduth added the Framework Issues related to broader framework topics, especially as it relates to javascript label Apr 24, 2018
@aduth aduth requested a review from youknowriad April 24, 2018 18:17

// Since query is optional for handled actions, avoid tracking empty object
// for default query on initialization.
withReturnUndefinedOnUnhandledDefault,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While these Higher Order Reducers are very clever, I wonder if they are too clever for maintainability :).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

withReturnUndefinedOnUnhandledDefault seems very Java to me (aka I have no idea what it does in reading it).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ended up removing withReturnUndefinedOnUnhandledDefault. It was needed because while ifMatchesAction is meant to allow the reducer to be run conditionally, we must still handle the initialization case. The other option is to provide a default value as an argument of ifMatchesAction, but this felt redundant when the reducer should be able to handle this on its own. Instead, in 623bad6 I remove support for optional query from getQueryParts and test for the property in the action for transforming via replaceAction. Thus, we get the same end-result.

perPage
);
},
requestingPageByPerPage( state = {}, action ) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think it's better to track the requesting state closely to the data or better in a separate "network requests" reducer. I've used mainly the first option for a long time but lately, I tend to think that the second one might be better as it's more generic and also speaks to the fact that these requests are "optional" in an offline mode.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think it's better to track the requesting state closely to the data or better in a separate "network requests" reducer. I've used mainly the first option for a long time but lately, I tend to think that the second one might be better as it's more generic and also speaks to the fact that these requests are "optional" in an offline mode.

I thought about this a bit and while I tend to agree that requests doesn't need to be tied so deeply to the items / queries, in practice if we try to separate them we'd end up duplicating much of the same logic anyways. As implemented, they're at least separate so far as being their own individual reducers (earlier iterations were combined into one). The behavior of getQueryParts relies on the fact that we generate a stable key which does not include pagination details. To track separately, we'd need a stable key with the pagination information. I see a few options here:

  • Manually concatenate page and perPage back into the stable key string in requests state
  • Replace getQueryParts with a simpler getStableKey, and for cases where we need to omit pagination details, do so before passing the object to getStableKey
  • Avoid building in pagination awareness into queried-data altogether. Just treat each query as distinct from one another, even on pagination arguments.

On reflection, the last of these is tempting, and could simplify things overall. The idea with the current implementation is that queries are considered the same ignoring pagination information.

There's also the question of: How much does "queried-data" make sense as a thing separate from the REST API? In your offline optional requests system you have in mind, would we still have concepts of querying by ?order=desc&page=2 ?

*
* @return {?Array} Query items.
*/
export function getQueriedItems( state, query ) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think this should be memoized to avoid creating a new array on each call?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One issue with memoization is the same as described at #6360 (comment) . For example, consider:

export default withSelect( ( select ) => {
	const { getCategories } = select( 'core' );

	return {
		categories: getCategories( { order: 'desc' } ),
	};
} )( CategoriesBlock );

In this case, we'd encounter a memory leak in that each object reference would be strictly non-equal.

The getQueryParts function gets around this by using the withWeakMapCache helper, which is designed to alleviate this by caching only as long as the object reference is in scope, otherwise allowing garbage collection to occur. Unfortunately that's only effective where we have a single object argument, not two.

We could perhaps keep a local WeakMap cache in scope, and do memoization manually.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we have a specific memoization here where we compare the first argument using strict equality (state) and the second one using deep equality (using EquivalentKeyMap maybe)?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we have a specific memoization here where we compare the first argument using strict equality (state) and the second one using deep equality (using EquivalentKeyMap maybe)?

Was able to get this idea working in c502dfc.

/**
* Higher-order reducer which returns undefined if the original reducer returns
* the default state value. Used in combination with `onSubKey` in avoiding
* assingment for undefined value to key.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had some trouble to understand the exact purpuse of this HoR even with the description here. I think it's the equivalent of a "pure" Higher Order Reducer right?

Copy link
Member Author

@aduth aduth May 4, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had some trouble to understand the exact purpuse of this HoR even with the description here. I think it's the equivalent of a "pure" Higher Order Reducer right?

While it's since been removed, the original purpose was not for purity, but rather to avoid the case where in state initialization, the reducer would be run, and since query was optional for getQueryParts, the initial state would end up looking like:

{
	"items": {},
	"queries": {
		"": {
			"itemIds": null,
			"requestingPageByPerPage": {},
		}
	}
}

...Where it would mistake the initialization as the default (empty) query.

*
* @type {string}
*/
export const TOGGLE_IS_REQUESTING = '@@queried-data/TOGGLE_IS_REQUESTING';
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any specific reason to introduce the constants. Not that I'm strongly against them but I do wonder about having some actions defined as constant and others not.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any specific reason to introduce the constants. Not that I'm strongly against them but I do wonder about having some actions defined as constant and others not.

It was implemented in the mindset of being an independent module utility, with prefixes on the action types to avoid potential conflicts. In earlier iterations, I had the consumer listening for the particular action types, where I didn't want them to be concerned with said prefixes. Since now only the internal reducer is concerned, we could just replace them with strings. Ultimately, I don't consider these files to be part of core-data or the data module patterns whatsoever, so the conventions don't overlap as much as it might appear on the surface.

Maybe a bit premature to modularize this, but having reimplemented effectively the same thing on many separate occasions, I was motivated toward something more universal 😅

Copy link
Member

@danielbachhuber danielbachhuber left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've tested this locally to confirm category UI still loads as expected. However, I can't comment to whether the abstraction is useful, though. It doesn't appear to address #5820

Copy link
Contributor

@youknowriad youknowriad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍 My comments are not really blocking, mostly questions

@aduth aduth force-pushed the add/core-data-pagination branch from c3d396e to c502dfc Compare May 4, 2018 16:06
@aduth
Copy link
Member Author

aduth commented May 4, 2018

Also, to make sure there weren't any large incompatibilities, I took a stab at what it'd take to port entities to using this approach. I think it'll be fairly straight forward, and the following patch gets most of the way there:

https://gist.github.com/aduth/0c69a1ee43cf07cbfaa0316e64186f2b

I'll not include this as part of this pull request, but as a proof-of-concept in anticipation of a follow-up.

@aduth
Copy link
Member Author

aduth commented May 4, 2018

Deliberating more on whether to flatten this into core-data or not, and to treat queries as distinct (including pagination), I recall expecting that queries will have more metadata that is intended to be shared across all pages of data. For example, total items. This was included in early iterations here, but dropped since it's not yet used (though I expect it will be needed).

On the point of tying requests state to items queries, at its most abstract the goal of such a tool is in accommodating behavior where only subsets of the total data associated with a given query may be known at a given time. The only thing not currently capable in a barebones implementation of this is satisfying the need to detect whether a resolver is pending. The data module already memoizes resolver fulfillment, so I'm wondering if we could leverage awareness of whether there's a next in the resolver generator and create a generic selector which operates on pending resolver state. That way, we wouldn't need to explicitly track requests at all. Let me see if I can come up with anything here.

@youknowriad
Copy link
Contributor

@aduth Hey Andrew, do you plan to revisit this one? That's one of the missing pieces to drop our current withAPIData usage.

@aduth
Copy link
Member Author

aduth commented Jun 12, 2018

Yes, I'd like to. I'll need to catch up on where I'd left off, and in what ways more recent changes would have posed conflict (largely #6462 and #6600).

@aduth
Copy link
Member Author

aduth commented Jul 17, 2018

I've unfortunately not yet found the availability to prioritize this one recently. If you'd like to pick it up, feel free. Otherwise, I'll plan to revisit once API and general writing flow has stabilized.

@youknowriad
Copy link
Contributor

I'm taking a look at refreshing this PR :)

@aduth
Copy link
Member Author

aduth commented Aug 2, 2018

Superseded by #8357

@aduth aduth closed this Aug 2, 2018
@aduth aduth deleted the add/core-data-pagination branch August 2, 2018 20:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Framework Issues related to broader framework topics, especially as it relates to javascript
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants